Papers with MLP transformations
Analyzing Memorization in Large Language Models through the Lens of Model Attribution (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing research has focused on extracting memorized content from LLMs or developing memorization metrics without exploring the underlying architectural factors that contribute to memorizing. |
| Approach: | They analyze how attention modules at different layers impact its memorization and generalization performance by using attribution techniques. |
| Outcome: | The proposed model can be used to mitigate memorization while keeping other components like layer normalization and MLP transformations intact. |